Papers with open-sourced VLMs
Red Teaming Visual Language Models (2024.findings-acl)
Copied to clipboard
| Challenge: | VLMs (Vision-Language Models) can be induced to generate harmful or inaccurate content through specific test cases. |
| Approach: | They propose a red teaming dataset which encompasses 12 subtasks under 4 primary aspects (faithfulness, privacy, safety, fairness) this dataset is the first to benchmark current VLMs in terms of these 4 aspects . |
| Outcome: | The proposed dataset shows that 10 open-source VLMs struggle with red teaming in different degrees and have up to 31% performance gap with GPT-4V. |
FoodieQA: A Multimodal Dataset for Fine-Grained Understanding of Chinese Food Culture (2024.emnlp-main)
Copied to clipboard
Wenyan Li, Crystina Zhang, Jiaang Li, Qiwei Peng, Raphael Tang, Li Zhou, Weijia Zhang, Guimin Hu, Yifei Yuan, Anders Søgaard, Daniel Hershcovich, Desmond Elliott
| Challenge: | FoodieQA is a manually curated, fine-grained image-text dataset capturing the intricate features of food cultures across various regions in China. |
| Approach: | They evaluate vision–language Models and large language models on unseen food images and corresponding questions. |
| Outcome: | The proposed dataset evaluates vision–language Models and large language models on unseen food images and corresponding questions. |